STA 141B PROJECT

DAVID WONG

This project explores more of the covidcast API data with the aim to explore how mask usage and other variables influence covid cases.

We will request the API data, for example, below to see the number of covid cases in the US.

import requests
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import sqlalchemy as sqla
import requests_cache
from my_func import req

a = req('fb-survey', 'smoothed_cli', 'nation', '20200406-20210601')

js = a.json()
datfram = pd.DataFrame(js['epidata'])
datfram.head()

	geo_value	signal	time_value	direction	issue	lag	value	stderr	sample_size
0	us	smoothed_cli	20200406	None	20201215	253	0.921000	0.026607	94222.0
1	us	smoothed_cli	20200407	None	20201215	252	0.882948	0.013821	336487.0
2	us	smoothed_cli	20200408	None	20201215	251	0.864801	0.010589	560878.0
3	us	smoothed_cli	20200409	None	20201215	250	0.844966	0.009094	746186.0
4	us	smoothed_cli	20200410	None	20201215	249	0.841315	0.008583	838676.0

More data will be collected.

b = req('usa-facts', 'confirmed_cumulative_num', 'nation', '20200406-20210601')
#requests.get('https://api.covidcast.cmu.edu/epidata/api.php?endpoint=covidcast&data_source=usa-facts&signal=confirmed_cumulative_num&time_type=day&geo_type=nation&time_values=20200406-20210601&geo_value=*')
# b = cumulative cases 
c = req('usa-facts', 'confirmed_incidence_num', 'nation', '20200406-20210601')
#requests.get('https://api.covidcast.cmu.edu/epidata/api.php?endpoint=covidcast&data_source=usa-facts&signal=confirmed_incidence_num&time_type=day&geo_type=nation&time_values=20200406-20210601&geo_value=*')
# c = daily new cases 
d = req('usa-facts', 'deaths_incidence_num', 'nation', '20200406-20210601')
#requests.get('https://api.covidcast.cmu.edu/epidata/api.php?endpoint=covidcast&data_source=usa-facts&signal=deaths_incidence_num&time_type=day&geo_type=nation&time_values=20200406-20210601&geo_value=*')
# d = daily new deaths 

js2 = b.json()
datfram2 = pd.DataFrame(js2['epidata'])

js3 = c.json()
datfram3 = pd.DataFrame(js3['epidata'])

js4 = d.json()
datfram4 = pd.DataFrame(js4['epidata'])

We will know plot the broad visualizations of the covid-19 pandemic cases and deaths in the US usnig matplotlib

import matplotlib.pyplot as plt
import matplotlib.dates as pltdates
import datetime

#date_begin = datetime.datetime(2020, 4, 6)
#date_end = datetime.datetime(2021, 6, 1)
#d_delta = datetime.timedelta(days = 1)
#dates = pltdates.drange(date_begin, date_end, d_delta)

dates = pd.date_range(start="2020-04-06",end="2021-06-01")

plt.subplot(3, 1, 1)
plt.plot(dates,datfram2.value)
plt.title("Covid-19 in the USA")
plt.ylabel("Cumulative cases")

plt.subplot(3, 1, 2)
plt.plot(dates,datfram3.value)
plt.ylabel("Daily cases")

plt.subplot(3, 1, 3)
plt.plot(dates,datfram4.value)
plt.ylabel("Daily deaths")

plt.rcParams['figure.figsize'] = (20,10)

We will now take a closer look at the demographics of each state and the cumulative number of cases and deaths up to the beginning of June 2021. This will be represented in a bar graph with the number of deaths below the number of cases.

e = req('usa-facts', 'confirmed_cumulative_prop', 'state', '20210601')
#requests.get('https://api.covidcast.cmu.edu/epidata/api.php?endpoint=covidcast&data_source=usa-facts&signal=confirmed_cumulative_prop&time_type=day&geo_type=state&time_values=20210601&geo_value=*')
#cumulative number cases prop
f = req('usa-facts', 'deaths_cumulative_prop', 'state', '20210601')
#requests.get('https://api.covidcast.cmu.edu/epidata/api.php?endpoint=covidcast&data_source=usa-facts&signal=deaths_cumulative_prop&time_type=day&geo_type=state&time_values=20210601&geo_value=*')
#cumulative number deaths prop

js5 = e.json()
datfram5 = pd.DataFrame(js5['epidata'])

js6 = f.json()
datfram6 = pd.DataFrame(js6['epidata'])

fig, ax = plt.subplots()
ax.bar(datfram5.geo_value, datfram5.value, 0.5, label = "cases")
ax.bar(datfram6.geo_value, datfram6.value, 0.5, bottom = datfram6.value, label = 'deaths')

ax.set_ylabel("Number per 100,000 population")
ax.set_title("Cumulative Covid-19 cases and deaths by state")
ax.legend()
plt.rcParams['figure.figsize'] = (20, 10)

From the above graph, we instantly see some states that stand out. The average cumulative proportionate number of cases per state is around 10,000. The states that are well below the average include: Hawaii, Maine, Oregon, Vermont, and Washington. The states on the opposite end include North Dakota, South Dakota, and Rhode Island. A state of interest that has an exceptionally noticable unusual proportion of deaths to cases is Utah.

We will look at leading vs lagging indicators. Some leading indicators include mask usage, bar and restaurant vists, and covid-like symptons. Some lagging indicators include cases, deaths, and hospital admissions.

First, the nation-wide values will be examined, then we will specify our analysis on groupings of states below average and above average. Utah will be examined seperately.

h = req('fb-survey', 'smoothed_wwearing_mask', 'state', '20200406-20210601')
#requests.get('https://api.covidcast.cmu.edu/epidata/api.php?endpoint=covidcast&data_source=fb-survey&signal=smoothed_wwearing_mask&time_type=day&geo_type=state&time_values=20200406-20210601&geo_value=*')
js8 = h.json()
datfram8 = pd.DataFrame(js8['epidata'])
datfram8.head()

	geo_value	signal	time_value	direction	issue	lag	value	stderr	sample_size
0	ak	smoothed_wwearing_mask	20200908	None	20201027	49	73.168143	4.430843	100.0000
1	al	smoothed_wwearing_mask	20200908	None	20201209	92	84.589826	2.149977	282.0058
2	ar	smoothed_wwearing_mask	20200908	None	20201209	92	79.519520	3.180487	161.0003
3	az	smoothed_wwearing_mask	20200908	None	20201209	92	84.110814	1.735139	443.8998
4	ca	smoothed_wwearing_mask	20200908	None	20201209	92	89.720539	0.707980	1840.0126

The data will be modified in order to create an interactive time series of the change in populations of mask-wearing across each state. If mask-wearing does have an infuence on covid cases, there should be some evidence for the change in number of cases as the mask usage changes.

datfram8['time_value'] = pd.to_datetime(datfram8['time_value'], format = '%Y%m%d') 

datfram8_mod = datfram8[['geo_value', 'value', 'time_value']]

#INTERACTIVE METHOD

import plotly.express as px


datfram_0 = datfram8_mod[['geo_value', 'time_value', 'value']]
datfram_0.rename(columns={'geo_value':'State', 'time_value': "Date", 'value': "Percentage wearing masks"}, inplace = True)

figu1 = px.line(datfram_0, x="Date", y='Percentage wearing masks', color='State')
figu1.show()

Overall, we see an increase in mask usage over time, with most states consistently having mask usage at 70-90%. We shall modify the data to see separately how the behavior changes in some of the outlier states incuding: North and South Dakota, Hawaii, and Utah.

df8_sd = datfram8_mod[datfram8_mod['geo_value'].str.contains('sd')==True] 
df8_sd.reset_index(drop=True,inplace=True)

df8_nd = datfram8_mod[datfram8_mod['geo_value'].str.contains('nd')==True] 
df8_nd.reset_index(drop=True,inplace=True)

df8_hi = datfram8_mod[datfram8_mod['geo_value'].str.contains('hi')==True] 
df8_hi.reset_index(drop=True,inplace=True)

df8_ut = datfram8_mod[datfram8_mod['geo_value'].str.contains('ut')==True] 
df8_ut.reset_index(drop=True,inplace=True)

fig, ax = plt.subplots()

ax.plot(df8_sd.time_value, df8_sd.value, label = "South Dakota")
ax.plot(df8_nd.time_value, df8_nd.value, label = "North Dakota")
ax.plot(df8_hi.time_value, df8_hi.value, label = "Hawaii")
ax.plot(df8_ut.time_value, df8_ut.value, label = "Utah")

ax.set_ylabel("Proportionate percentage of people who wore masks")
ax.set_title("Specific States")
ax.legend()


#plt.rcParams['figure.figsize'] = (20, 10)

<matplotlib.legend.Legend at 0x20d9b69ef70>

We now will construct the data to create a time series of the proportionate number of new cases per day by state.

#datfram8_mod
dates_8 = pd.date_range(start="2020-09-08",end="2020-11-18")
#datfram8_mod.time_value

k = req('usa-facts', 'confirmed_incidence_prop', 'state', '20200406-20210601')
#requests.get('https://api.covidcast.cmu.edu/epidata/api.php?endpoint=covidcast&data_source=usa-facts&signal=confirmed_incidence_prop&time_type=day&geo_type=state&time_values=20200406-20210601&geo_value=*')
js11 = k.json()
datfram11 = pd.DataFrame(js11['epidata'])

import plotly.express as px
datfram11['time_value'] = pd.to_datetime(datfram11['time_value'], format = '%Y%m%d')

datfram11 = datfram11[['geo_value', 'time_value', 'value']]
datfram11.rename(columns={'geo_value':'State', 'time_value': "Date", 'value': "Prop. New Cases per day"}, inplace = True)

figu = px.line(datfram11, x="Date", y='Prop. New Cases per day', color='State')
figu.show()

With a visualization of which states have cases been spiking, we have an idea of which states to look out for. Now we will construct scatter plots to compare mask usage with proportionate number of covid like cases in the community.

l = req('fb-survey', 'smoothed_hh_cmnty_cli', 'state', '20200908-20201118')
#requests.get('https://api.covidcast.cmu.edu/epidata/api.php?endpoint=covidcast&data_source=fb-survey&signal=smoothed_hh_cmnty_cli&time_type=day&geo_type=state&time_values=20200908-20201118&geo_value=*')
js12 = l.json()
datfram12 = pd.DataFrame(js12['epidata'])
datfram12.head()

	geo_value	signal	time_value	direction	issue	lag	value	stderr	sample_size
0	ak	smoothed_hh_cmnty_cli	20200908	None	20201119	72	19.675325	1.433581	769.0000
1	al	smoothed_hh_cmnty_cli	20200908	None	20201119	72	27.241248	0.691160	4149.1054
2	ar	smoothed_hh_cmnty_cli	20200908	None	20201119	72	26.410896	0.825067	2855.0818
3	az	smoothed_hh_cmnty_cli	20200908	None	20201119	72	16.219975	0.488709	5689.7096
4	ca	smoothed_hh_cmnty_cli	20200908	None	20201119	72	14.696445	0.233218	23049.1767

We will modify the dataframes to plot mask usage and covid-like cases in the community instead of more time series.

datfram12['time_value'] = pd.to_datetime(datfram12['time_value'], format = '%Y%m%d')
datfram12 = datfram12[['geo_value', 'time_value', 'value']]
datfram12.rename(columns={'geo_value':'State', 'time_value': "Date", 'value': "Percentage of people who know someone is sick"}, inplace = True)

datfram12.head()

	State	Date	Percentage of people who know someone is sick
0	ak	2020-09-08	19.675325
1	al	2020-09-08	27.241248
2	ar	2020-09-08	26.410896
3	az	2020-09-08	16.219975
4	ca	2020-09-08	14.696445

import warnings
warnings.filterwarnings('ignore')
datfram8_mod.rename(columns ={'geo_value':"State", "time_value": "Date", "value": "Prop. Percentage wearing mask"}, inplace = True)

datfram_1 = datfram12.merge(datfram8_mod)
datfram_1.head()
dff = datfram_1.iloc[:45]
dff2 = datfram_1.iloc[-29:]

figu3 = px.scatter(dff, x="Prop. Percentage wearing mask", y='Percentage of people who know someone is sick', color = "State", title = "Mask usage vs people sick in community for September 8, 2020")
figu3.show()

figu4 = px.scatter(dff2, x="Prop. Percentage wearing mask", y='Percentage of people who know someone is sick', color = "State", title = "Mask usage vs people sick in community for December 11, 2020")
figu4.show()

We see that there are obvious negative correlations between wearing masks and covid-like cases. This will be further analysed with spearman rank coefficient tests.

datfram_1.corr(method = "spearman")

	Percentage of people who know someone is sick	Prop. Percentage wearing mask
Percentage of people who know someone is sick	1.00000	-0.66816
Prop. Percentage wearing mask	-0.66816	1.00000

dff.corr(method = "spearman")

	Percentage of people who know someone is sick	Prop. Percentage wearing mask
Percentage of people who know someone is sick	1.000000	-0.742424
Prop. Percentage wearing mask	-0.742424	1.000000

dff2.corr(method = "spearman")

	Percentage of people who know someone is sick	Prop. Percentage wearing mask
Percentage of people who know someone is sick	1.000000	-0.563054
Prop. Percentage wearing mask	-0.563054	1.000000

By conducting spearman correlation tests, we see that rho is calculated around -0.74, so there is a relatively strong negative correlation between the two variables: the percentage of people wearing masks and people who know someone else who is sick.

For our final analysis, we will merge the dataframes with other signals from the fb-survey, with those signals being the percentage of people who, in the last 24 hours, visited restaurants/bars and those who attended large events with 10+ people. Then spearman correlation tests will be done.

m = req('fb-survey', 'smoothed_wrestaurant_1d', 'state', '20200908-20201118')
#requests.get('https://api.covidcast.cmu.edu/epidata/api.php?endpoint=covidcast&data_source=fb-survey&signal=smoothed_wrestaurant_1d&time_type=day&geo_type=state&time_values=20200908-20201118&geo_value=*')
js13 = m.json()
datfram13 = pd.DataFrame(js13['epidata'])


n = req('fb-survey', 'smoothed_wlarge_event_1d', 'state', '20200908-20201118')
#requests.get('https://api.covidcast.cmu.edu/epidata/api.php?endpoint=covidcast&data_source=fb-survey&signal=smoothed_wlarge_event_1d&time_type=day&geo_type=state&time_values=20200908-20201118&geo_value=*')
js14 = n.json()
datfram14 = pd.DataFrame(js14['epidata'])

datfram13['time_value'] = pd.to_datetime(datfram13['time_value'], format = '%Y%m%d')
datfram13 = datfram13[['geo_value', 'time_value', 'value']]
datfram13.rename(columns={'geo_value':'State', 'time_value': "Date", 'value': "Percentage of people who went to a restuarant in the last 24 hours"}, inplace = True)

datfram14['time_value'] = pd.to_datetime(datfram14['time_value'], format = '%Y%m%d')
datfram14 = datfram14[['geo_value', 'time_value', 'value']]
datfram14.rename(columns={'geo_value':'State', 'time_value': "Date", 'value': "Percentage of people who went to a large event (10+ people) in last 24 hours"}, inplace = True)

datfram_2 = datfram13.merge(datfram14)
dff3 = datfram_2.merge(datfram_1)
dff3.corr(method = "spearman")

	Percentage of people who went to a restuarant in the last 24 hours	Percentage of people who went to a large event (10+ people) in last 24 hours	Percentage of people who know someone is sick	Prop. Percentage wearing mask
Percentage of people who went to a restuarant in the last 24 hours	1.000000	0.857964	0.324240	-0.720612
Percentage of people who went to a large event (10+ people) in last 24 hours	0.857964	1.000000	0.496953	-0.834597
Percentage of people who know someone is sick	0.324240	0.496953	1.000000	-0.668472
Prop. Percentage wearing mask	-0.720612	-0.834597	-0.668472	1.000000

From examining other leading factors, we see that there is a positive correlation between going to a restaurant or large event to knowing someone in the community is sick. There is a strong negative correlation between wearing a mask and going out to events and restaurants.